The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), has proven useful to model the as making noisy-rational choices, with a "rationality coefficient" capturing how much noise entropy we expect see in behavior. Prior work typically sets rationality level constant value, regardless of type, quality, feedback. However, many settings, giving one type feedback (e.g. demonstration) may be more difficult than different answering comparison query). Thus, less depending on In this work, advocate that grounding coefficient real data for each rather assuming default significant positive effect learning. We test both simulated experiments and user study find overestimating can have dire effects learning accuracy regret. also fitting enables better learning, even when deviates significantly choice due systematic biases. Further, affects informativeness type: surprisingly, demonstrations are not always most informative---when acts very suboptimally, comparisons actually become informative, is same both. Ultimately, our results emphasize importance advantage paying attention assumed human-rationality-level, especially agents actively learn multiple types